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ABSTRACT 

Presented' is a plan for evaluating Project KIDS 
(Kip-dling Individual Development Systems) , a model program for 
developmentally delayed and physically handicapped infants, 
preschool children, and their families. The KIDS program is 
as providing services in a home-based training program, 
infant stimulation classes9 and school-based early 
units. Evaluation methodology is explained and evaluation questions 
are listed and detailed for each of the following program components: 
children component, parent coaponent, staff development component, 
and demonstration/dissemination component. Brief descriptions are 
provided of each of the specified measirement instruments, such as 
the^ Bayley scales of Infant Development, McCarthy Scales of 
Children's Abilities, KIDS Inventory of Development Scale, KIDS 
Parental Expectations Test, and the KIDS Staff Development Survey. 
Data collection schedules and reporting schedules are included, 
(IM) 
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INTRODUCTION 

Project KIDS (Kindling Individual Development Systems) is a 
Dallas Independent School District model program for handicapped 
infant^s^, toddlers, preschool children and their families. The program 
is financially supported by the Bureau for the Education of the Hanii- 
capped in the U» S« Office of Education and is a part of a nationwide 
network of early childhood projects for the handicapped. Supplementary 
services and consultation are being provided by the University Affiliated 
Center and the Special Education Program oC" the University of Texas at 
Dallas. , 

The project serves developmetitally delayed and physically .handi- 
capped preschool children from 0-60 months of age: Services aire 
delivered in a home-based training program, center-based infant stim- 
ulation classes, and school -based early childhood class units.- Project 
KIDS attempts to integrate the child's parents into the instructional . 
role in cooperation with project staff. In conjunction with parent 
involvement, a continuum of activities are available to parents to 
assist in promoting fparenti'ng knowledges and skills in the areas of 
developmental^ del ay and physical handicaps. 

The project instructional staff consists of two home-based teachers 
and two teacher aides and one school-based teacher and aide (during the 
1976-.77 school year, there will be four school-based teachers and 
aides). Staff resources also i.nclude appraisal and curriculum consul- 
tation from tht "niversity of Texas at Dallas and ^extensive appraisal 
services from- the University Affiliated Center. . 



Evaluation Methodology 

The following sections. of this document provide a brief general 
overview of operating procedures within each project component and 
outline the evaluation questions and the design and euialy^is procedxxres 
for each component* The design and analysis sections describe .data 
collection procedures and a^iticipated statistical analyses, both 
, descriptive and inferential statistics, if appropriate.. In many 
cases, the analysis section also includes a discussion of the kinds 
and usefulness of the anticipated evaluation results* 

The evaluation questions designated for, each comp/.Tient represent 
. information priorities established by project managenient in cooperation 
with 'evaluation personnel. The evaluation questions are basic to the 
evaluation effort in that they determine the chava^er of the evaluation, 
and consequent methodology. One advantage of the evaluation questioji 
is that it:. may be related directly to project objectives and thereby 
focus the evaluation toward project 'objectives. In one sense the 
evaluation question Serves as a conceptual bridge^ between project 
objectives and evaluation. As one might expect, the specified evalua- 
titjn questions do not usually reflect all desired information but 
represent a trade-off among information apriorities, evaluation 
resources, and project characteristics. 

An instrumentation ^section is included to describe the measurement 
. and observation devices to be used in data collection in all components 
and to give available psychometric data regarding selected instrumenta- 
tion. The measurement and reporting schedules are included to provide 
a time framework to assist in implementating the evaluation plan within 
all components. 

ERIC ^r^ 



Children Component 

Project Procedure . Referral of children to Project KinS inay 
come, about through several sources, some of which *are community 
agen>:ies, the DISD, parents, hosfyitals, and pediatricians*. In order 
to be eligible for the project, the following criteria must be met: 
' " 1. the parents (or primary caregiver) must reside within 
the DISD. 

2. the child must be between six and sixty months of age. 

3# the child must be .developmentally delayed (primarily mentally 
retarded) as determined through formal intellectual assess,- 
ment (IQ must be two or more standard deviations below mean)# 

4. the parents must be willing for the child to be enrolled in 
the project. 

. 5. the parents must be willing to devote approximately an hour 
daily to intervention procedure's. 
Once a child is referred to the project, initial screening consists 
of measurement witih the Denver Developmental Screening Test and a 
.^^general physical examination. If the extent of delay is significant, 
a comprehensive, assessment .is initiated with the Bay ley Scales of 
Infant. Development being administered to obtain a furthef^ measure of 
developmental delay. The McCarthy Scales of Children's Abilities is 
used to assess development fl^r those children who exceed the age 
range of the Bayley. , 

The purpose. of the comprehensive assesanent is to obtain d^ta from 
an inter-disciplinary appraisal team. Project assessment procedures 
also yield data which meet eligibility criteria of the Texas Education 



Agency for Special Education programs* 'Much of the effort toward 
establrishing a working relationship with parents and obtaining 
family background data also takes place during this oomprehensive 
assessment phase* > 

-f ' ' ' ' • 

Once project -eligibility is established, the ciiild is enrolled 

in the project and Jthe KID§ inventory of Development^^^p/adb^ 

to provide a baselixie for instructional programming* Three basic 

administrative arrangements (or treatment populations) f orm ^the 

framework' for the organization and delivery of services to children* 

The youngest children (starting with 6 - 12 montJas of age) receive 

service through a home-based arrangement (designated as population A)* 

* . o » 

Project staff visit the home twice weekly and assist the parent in . 
implementing the planned instructional activities • Older children 
(starting with about 24' months) attend weekly infant stimulation , 
classes held at the project center and.^lsp receive home--f6r?b(^p 
vlisits to support the .center-based instruction (de'signatiad, as population 
Children who are three years of age attend early childhood classes at a 
re^ar DISD ^chool site (designated as population C)* 

The primary purpose of service to children is to support) 
patterns of development within each child and to promote esp^ially the 
-areas of delayed development • The basic instructional vehi(:le is the 
Mini Activity Plan (MAP) which specifies t^e area -of ^ delay, purpose of 
the lesson, behavioral objective, materials needed, step-by-step 
instructional procedures, and a criterion level for completion of the 
MAP* The MAP also contains a progress chart where parents chatt daily 
performance of the child. Following each home visit, project staff 
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campiete a record of the home visit in tems.of parent participation, 
attitude, general atmosphere and so f or th« / However, parents of school- ' 
based children do not receive regular home follow-up. The content 
and sequence of the MAPs for each child are largely a ftinction of 
performance on the KIDS Inventory of Development. While the number 
of MAPS completed and time per MAP will vary across children, the 
anticipated completion time for most MAPs is one week. 
Evaluation Questions . 

1. How many childreri were served in Project KIDS? 

2. What demographic characteristics described the population of 
childjren served? ^ 

3. What was the instructional history of children served? 

4. What were the areas of delay specified in the MAPs? 

5. What. were, the instructional objectives specified in the MA^s? 

6. Wiat was the extent of iristructioa received by children? 

7. What was the observed improvement, if any, in the areas of 
developmental delay? 

8. -What time constraints were needed for children to master the 

instructional objectives? 

9. To what extent was observed impro/Zement due to intervention 
. . through Project KIDS? * 

10. What was the instructional history and progress of children 
served in Project KIDS over a three-year followup period? 

11. What vi^ia t:Lfe cost of the children component? 

Design a nd Andlvsis. The following details evaluation methodology* 
for each of the above questions. 



1. How many children were served In Project KIDS? 

The answer to question one will take the form of a simple taJ)uLation 
of students served in Project KIDS. The tabulation will be broken- 
down according to the kind of administrative arrangement (i.e., home- 
based^ center-base<i> or school-based). 

2 . What demographic characteristics described the peculation of- 
children served? 

The primary purpose of the above question is to establish a fairly 
complete description of children served. Routine project assessmerAt 
procedures result in the coljLaction of 'consifderable relevant da.ta on 
each child serve<i* in the project. In order to organize these data for 
easy tabulation and accessibility, a record layout for coding, of informa- 
tion will be developed. Children will be given a project identification 
n-^inber, and. all' critical information can be stored in machine readable 
form. This procedure will greatly facilitate future longitudinal 
research and evaluation efforts. 

In terms of more immediate infomation needs, the reapcnse to 
question two will provide one basis for interpreting p:;oject ef f ejctive- 
ness and for planning and possible project- modification. Additionally, 
the information could be helpful to others involved in similar kinds . 
of projects. Descriptive statistics conputed will include" the percent 
of children relative to age, race, it.i sex. Analysis will also provide 
a descriptive summary of chi'^d. ^^n L.gjed on the comprehensive assessment 
results as these relate to developmental delay and physical handicap. 
3. What was the instructional history of children served? 
The response to. question three provides very important information 
relative to interpreting project effectiveness, since the expected 



extent of Improvement is a function of the exteftt of treatment 
(i.e4^, instruction) received in the project. The question also 
prayi-das inf oniiation helpful in future planning for delivery of 
services- and ii replication at other sitei. 

Data analysis will consider instructional contact time by 
project staff and by parents as dependent variables. Descriptive 
statistics for the treatment populations A, B, and C will -include ' <^ 
average contact time and "the range in contact time. Analysis will 
also take into account the relative success or effectiveness of . 
parent-staff sessions as perceived ^nd reported by project staff. 

4. What were the areas of delay specified \n the MAPs? 

The source of information for responding to question f ou?: "Ill ' 
be the areas of delay identified on the MAPs for each child at the 
beginning of instruction within the project. Analysis will/ consist 
of a simple frequency count for each of the four areas of delay 
contained in the KIDS Inventory of Developmiant (gross motor^ fine 
motor, cognitive, an^ self-help) . ^ » 

x^alysis will ai30 consider the priority ranking of the areas 
of delay '^S' identified in the assessment of each^ child. The average 
rank of each area across children will be computed, and for each area 
of delay, the percent of children whose assessment shows a ranking ol: 

one (i.e.; top priority) will also be computed. 

/ 

5 • What were the instructional objectives specified in the MAPs? 

The information sotirce for question five will be the population 
of objectives specified on all MAPs for all children. Objectives will 
be tabulated according to the four major areas within the KIDS Inventory 



of Development (gross motor, fine motors cogi^itive/ Language, and 
self-help) And grouped i^ccording to adaini£»trative arrangement 
il.e.f treatment population). 

6. What was te^ extent of I n stiructlon received by children? 
The two dependent varlajles >:onsldered will be the number and kind 
of objectives on the MAi's the extent of ^Instruction delivered by 
parents. The source of iriicrmatlon for the extent of parent Instruction 
will be the' charting records malntalne|i by' parents* While it ^s assijmed 
that these records are not completely error .free, it^ris thought: chat 
they will provide a reasonable indication of the extent of instruction 
delivered by parents (input from project staff will augment these records); 
J]^he response to question six pertains most importantly to the interpre- 
tation of project effectiveness relativia to progress of individual 
children. Analysis will consist of a tabulation of^ objectives for each 
child and computation of the number of instructional sessions presented 
by parents and by staff, for each child. - y 

7, What was the observed improvement, if any, in the ^areas of 
developmental^ delay? y'^ 
Questions one through six largely address issues relatjpd no 
program implementation and operation, and these questions constitute 
the bulk of process evaluation in the children component. Question 
seven addresses the issue of project effectiveness. The first step in 
evaluating project effectiveness is determining whether or not children 
served in tlxe project made any progress. The second step is to determine 
if the observed progress, if any, is attributable to Project KIDS. 
(Question nine addresses this issue.) 
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The dependent variables in the analysis for question seven will 
- W the nijmber of objectives mastered, scores on the KIDS inventory of 
Developtaent, ,ahd scores on the Baylay Scales of Infant Development and 

McGartay Scales, of Childrerf s Abilities. Only toe perceptual-nerf ormance 

■ * ■ \^ * , 

^iv.d -tiiov:or scales of the McCarthy C^ll be considered in responding to. 

this question. The Bayley and McCarthy will be the instruments Of choice in 

estimating the possible Impact o/ Project KIDS on children's develojpmant* 

These instruments were selected because ^f relevance to project goals 

and sound psychometric characteristics. Any children who experi^^ncad 

significant trauma during the measurement period (such -as extended ' 

surgery, severe faully disruption) will be excluded from janalysis. 

Analysis for the number of objectives mastered wlll.be a simple 
tabulation of mastered objective? for each child at the end of the 
measurement period (Juile 1, 1976 through May 31, 1977). The basic 
measurement schedtile for the KIDS Inventory, Bayleyj and McCarthy 
will be a pre-post framework, where the pretest takes pl^ce at entry 
into the project (after June 1, 1976) and posttesting takes place 
12 months after the protest. The three scores from the Bayley (mental, 
motor, and behavior rating) will be considered separately in the " • ' 
analysis, as will the two scores from the McCarthy. 

.The primary emphasis in responding to question seven will be 
placed on analysis of pre-ppst scores from the Ba^'ley and ^McCarthy. 
The KIDS Inventory of Development has not yet undtergone extensive field 
testing, and the usefulness of the inV^ntoiry in c jsessing pre-^post 
changes is not known at this time^ Projections call for a criterion- 
referenced perspective in scoring and analysis. It is expected that 

11 
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xaastery of objectives and performance on the Inventory will* be 
moderately or hlglily related. Disagreement between these two 
dependent variables may alert project staff to a possible breakdown 
In 6he basic relationship between the inventory and educational 
progransoing in the MAFs. 

It is probable that scores from the KIDS Inventory wil^l be 
moderately related to objective mascery, but the inventory is still 
in a developniental state. Consequently, the primary emphasis in 
responding.no thii^i question will be placed on analysis of pre-post 
scores frotr. the Bayley and McCarthy. The primary analysis will 
consist of the computation of average scores at time* of pretest and 
at time of posttest. A repeated-measures ANOVA will be used to 
generate an F-test of the significance of any pre-post change. 
Appropriate ranges and standard deviations \fill be provided as well. 

Unfortunately the assessment of pre-post change in projects 
such as KIDS is s'ubject to a. bothersome limitation known as statis- 
tical regression to the mean (i.e., average). This can happen 
.because Project KIDS serves children wjfo make up the extreme end 
of the population of all children. Even though one might, think of 
development ally delayed children as an identifiable population in and 
of themselves, they still make up the lower end of the total populati 
of children. In fact, this is why they are selected, and it is this 
very fact that dan result in the statistical regression to the mean 
effect. For a fairly lucid explanation of this statistical regressio 
phenomenon, the reader is referred to Campbell and Stanley (1963, 
pp. 10-12). The net result of a regression effect will b6 a higher. 
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posttest Chan pretest score (given selection for Low pretest scores), 
even though there may have been no pre-pust change in truie scores* 
Obviously, one must find a way of controlling the regression 
to the mean effect, and the most preferred procedure would be to 
assign children randomly to control, and treatment groups and to 
collect pre-post data in both groups* In that case, one would expect 
Che effe-it of regression to be the same in both groups and observed 
differences between the two groups could be attributed to project 
intervention^ However, in first chance intervention projects, 
control groups are extremely difficult (if not impossible) to 
obtain. One is faced with the ethical issue of denying treatment 
to the control group and with the great difficulty of locating 
sufficient number of children to make up the control group. 
^ Another way to control for the regression to the mean effect is 

to use one instrument for the selection of children into the project 
and a second instrument for pra-post measurement. The basis for this 
control is that the selection instrtment scores of children selected 
for the project contain a pre-ponderance of negative measurement 
error, since these children wjare selected because of their extreme 
negative scores (relative to the total population of all children) . 
If one were to administer the selection instrument as a posttest, 
many or most of the ch:?.ldren would likely have positive rather than 
negative error in the posttest score. It is this reversal of error 
that results in the regression to the mean effect (Ln cases where the 
first measurement contains a preponderance- of either negative or 
positive error). However, if pretest scores are from a second 
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Instnuneut and not the selection instriinent, there is no reason to 
expect a preponderance of either negative or positive error in pretest 
scores, since the selection process did not "contaminate" the pretest, 
scores . 

A3 stated previously, Project KIDS uses the Denver Developmental 
Scale as a screening or selection instrument, and the Bayley Scales of 
Infant Development and McCarthy Scales gf Children's Abilities will 
provide pre-post measvurements. Children in the school-based population 
may be selected according to District criteria for special class place- 
ment, but these criteria will not include scores from the McCarthy. 
Hence, there is no reason to expect any observed pre-post changes in 
Bayley scores to be confounded with a regression to the ,mean effect. 

Another means of assessing the observed pre-post improvement will 
be to reference the Bayley and McCarthy scores to published test norms. 
The percentage of scores at certain levels will be computed in comparison 
to the norms (e.g., 60% were one standard deviation below the mean, 30% 
were two standard deviations below, etc). The percent differences aft 
these levels also will be calculated from pre to post obsen-atiorg. The 
Bayley and McCarthy norms provide th- average score plus and minus three 
standard .:*eviatious ; any scores below that level would have to be 
emitted from analysis. 

In addition, reference to- test norms can provide an estimate of 
normal development in terms of i:he Bayley and McCarthy over a 12-month 
pre-post period for the average child at a given age. Comparison of 
pre-post gains of children in the project to gains of nonnative children 
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may provide a useful point of reference for project staff, but the 
utility of such comparisons is limited. Comparison to a normative . 
popxilation will also provide some measure of the potential of children 
for social participation, i.e., a measure of how well project KIDS • 
children are doing in relation to normal children. Social participation 
of children in terms of parent and professional expectations is also 
considered under question eight of the Parent Component section of this 
doctiment. 

Video recordings of children's performance will also be collected 
on a^pre-post schedule. However, at the time of this writing, work in 
video recording technology is just getting under way in the project, 
but it is expected that evaluation methodology within the next year or 
two will include video recordings as an important data source. Pre- 
liminary evaluation efforts toward this end will include pre-post 
observations on a random sample of children in the project. A behavior 
rating scheiAfe will be developed or adopted to quantify the observations. 
Analysis will also include computation of the range and standard 
deviations associated ^th the averages outlined above. 

8. Wh^it time constraints were needed for children to master 
the Instructional bbj actives ? 
Project staff^'will determine whether or not a child has mastered 
each objective by administering the perfonftance task specified in 
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the MAP. Each M/^ specifies a mastery criterion so that the deter- 
mination of mastery wlLL bia relatively consistent across staff 
mjdabers. Parents are cxpecte'^d. to chart: performance data on a daily 
basis, but it is thought that tba extent o|, 'measurement error tn 
tjaase data would be too great for inclusion in evaluation analyses. 
The dependent variable will be the number of days from the time 
work on the object./ Vit began to the t:^ of mastery as determined by 
project staff. 

Data analysis for question six will only consider those 
objectives completed by, May 31, 1977 .and will consist of the compu- 
tation of the average .time for each child to master objectives with- 
each of the four areasv of the KIDS Inventory of Development and will 

also consist of the Computation of the average ti'me across children 

^' * • * ■ 

for mastering objectives within each area of development. The- 

diagram below outlines the comp,utationai:procedure. 

A word of caution is in order since project staff -^make no specifi 

attempt to make objectives comparable across children or areas of 

development or even xd.thin selected developmental sequences within 

children. Consequently, interpretation of mastery time^must consider 

this limitation and bear in mind that certain comparisons, such as 

be^:ween children or developmental areas, may not be meaningful. 
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*Note that objectives will not neceasarily be Identical across children. 
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9. To what extent was observed Improvement due to intervention 
through Project: KIDS? 

The ideal response to question nine would be data from a research 
study based upon the random assignment of children to Project KIDS 
and a control condition of no intervention. However, it is almost 
impossible to obtain a control group in projects such as KIDS (see 
discussion under question seven). Nevertheless, if improvement in 
children does take place, one would want to know if it were due to 

project intervention or to an extraneous factor, perhaps just 

V 

' matu^jation. (The issue of a regression to the mean effect has been 
addressed under question seven). 

i ^7 
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In the absence of a control group or .even a comparison group 
of children, one must find an alternative point of reference for 
comparison. One aplproach to obtaining a comparative point pf * 
reference has been to calculate an expectancy score based on pre- 
test performance. The basic procedure is to compute a develop- 
mental rate as the ratio of developmental age to chronological age 
at the time of pretest. ^ Simeons son and Wiegerink (1975) l?uilt on 
this concept in suggesting an index of efficiency for cciV'pcring c 
heterogeneous children within a project or across projects. 

Another approach to obtaining a "'comparison pqint of reference 
,is to adoptr a time lag design and form a comparison group or groups 
within the project as a function of age at time of ,pre-post measure- 
ment. The basic premise of this time lag approach is that the pre- 
test scores of one group can serve as the comparison for posttest 
scoires of another group. For a discussion of time lag design, 
see Goulet (1975). 

Another popular solution is the use of a matched pairs design. 
Macy and Carter (1975) hava reported success in matching severely handi 
capped school age children, but good success in matching requires a 
large population from which to draw. All of the above ^^^oaches have 
limitations which restrict their application to the problem of a com- 
parison point in project KIDS. Consequently, the evaluation uses 
a theoretical control which consists of a projections made by a 

/ • ' • ■ . ^ .... 
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panel of experts abc^itT'posttest performance of children in the 
project. Panels of experts are conmonly used in instrument 
validation studies, but there has been only limited use -of experts i 
in developing theoretical controls. 

The theoretical control approach s^ems. to be especially^appli- 
cable to Project KIDS .in that, th^ project addresses developmental 
delay, an area with sufficient background research and clinical^ 
experience to make projections possible. The procedure will be to 
select six' experts in child development from the Dallas are^. The 
expertSLJri.ll be totally Independent of Project KIDS and will 
possess exteasive credentials and experience in child development. 
Each expert will indepently review the complete assessment records 
of each child (names of children will be withheld) and will then 
formulate an assessment profile of the child projected forward 12 
months from date of pretest assessment. Experts will base their 
projections on the assumption of no intervention^ In making 
projections, experts will consider the total family environment and 
not just the child. 

Analysis will focus on the e,xte'^>t: of agreement between the 
projections of experts and actual posttest measurements (objective 
mastery and the Bayley and McCarthy). At the time of this writing, 
the details of analyzing the extent of agreement have not yet been 
fully developed, but these procedures will be described in the final 
evaluation report. Limxtatiotis of the theoretical control approach 
" appear to depend on the confidence one can plar.e in the projections 
of the panel of experts and in the statistical method of assessing 

19 
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aaxeement between actual posttast scores and projections. A 
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research study will be conducted within the evaluation effort^4,n 
order to address. these limitations^ Curtis and Donlon (1972)' 
have addressed the issue of agreement among experts in rating video 
tape recordings of multiply 'handicapped children^ 

' 10# What was the three^vear instructional history and ptogress 
of children served in Project KIDS? 
: The purpose of including question ten is primarily to emphasize 
the need for longitudinal information and to ensure planning for the 
implementation of longitudinal research within the project' evaluation^ 
The response to this question in the 1976-77 evaluation should be the 
planning and implementation of longittSdinal data record keeping 
procedures. Even though the current funding source for Project KIDS ^ 
is. limited to a relatively small number of years, longitudinal study 
of children served in the project should be ensured as a contribution 
to the field. 

11, What was the cost of the children component? 
The basic analysis in response to question eleven will consist 
of the computation of the cost of delivery of services' to children 
in terms of project staff salaries, materials, and -adjunct services. 
Cost per child will be figured for selected key points in project 
service delivery, such as screening, appraisal, and so forth. Analysi 
will -also include the computation of average costs and the range in 
costs. Project staff will maintain appropriate cost records in terms 
of project components. Records will also be kept for a ccxnbined 
grouping of components, since there wi\l be costs which are not 
readily identified with any one project component. 
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ObvlousLy, cost accounting' is a compLek area, 'and the cost 
of the project could and should be studied in depth. However, 
the current evaluation considers only basic cost analysis/ It 
was thought that at the cxirrert time, the most expedient use of 
evaluation resources was to focus on the kinds of information 
specified in the previous questions. 

'f 

Parent Component - ♦ ' 

Pr elect Procedure. The project involves parents of children - * 
in several basic ways. Perhaps the most obvious is the utilizatitm 
of parents in a primary instructional capacity, but parent involve- 
ment alsj includes conferenceis with project staff, meetings with 
Other parents, and cooperative plans for individual parent training 
activities. ' 

Once a child is enrolled, project staff work closely with the 
parent on a one-to-one basis in order to establish rapport ahd to 

•■ . ■ -( 

train and assist parents in the instructional activities specified 
in the MAPs. Par5l^nta^ of children in the home-based treatment popula- 
tion receive two w^kly home-visits from proje'ct staff, and those In 
the center-based population receive one weekly home visit. The 
school-based population does not currently involve regular heme 
followup, but school based teachers attempt to involve parents in 
the children's instruction at home by means of l:he MAPs. ' 

In addition to assuming, instructional responsibilities^ parents 
also have the opportunity for improvement of their painting skills 
through cooperative plans of parent training activities. These 
plans are cooperatively developed with project staff and are based o 
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a self -report inventory of parenting needs. Parents select areas 
of felt strength and weakness on a self-appraisal inventory, which 
covers such things as management . techniques, basic concepts in mental 
deficiency, nutrition, etc. The cooperative plans of parent training 
activities are individualized and may include a variety of activities 
such as visiting another parent, viewing a film, reading magazine 
articles, and so forth. The cooperative plans are then translated 
into MAPS for parents to facilitate implementation of the plans. 
. Evaluation Questions. 

.1. What, demographic characteristics described the parents 
of children in Project KIDS? 

■ ■- " ' 

2. What were the needs reported by parents in the self- 
appraisal inventory? 

3. What was the extent of agreement between parenting needs 
reported by parents on the self -appraisal inventory and 
parenting needs of parents as seen by project staff? 

4. What were the objectives specified in MAPs for parents? 

'I 

'I 5. What was the success rate in completing MAPs? 

6. VTnat progress was made in meeting identified needs of 
parents? 

7. What was the reaction of parents to the project? 

8. What was the extent of agreement between parental expectations 
of the child and expectsuions of the child as judged realistic 
by ^professional personnel? ^ 

9. What were the observed changes in parental needs over a three 
• year period? 

10. What was the cost of the parent component? ■ ^ 

22 
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Design and Analysis . The following details evaluation methodology 
for each of the above questions. The term parent Is usually understood 
to refer to the parent who prixnarily interacts with the project. This 
will likely be the^mother or female gxiardian. 

1. What demographic characteristics deyjcribed parencs of children 

in Project ^ KIDS? 
Analysis will consist of a descriptive summary the parent 
■^^population in terms of relevant socio-economic variables. (Question 
one will attempt to consider the family, not just a singular parent.) 
^ 2. What were the needs reported by parents in the self -appraisal 

^ inventory? . ^ 

The basic procedure will be to list the population of needs reported 
by parents in the- Parenting Self-Appraisal Inventory. If possible, the 
needs will be grouped and ranked according to the frequency the need or 
group of needs was cited by .pareficsi^as either a strong or weak area. 
Data will be useful 'in viewing the/4xteht of common and unique needs 
perceivfid by parents* 

3. WhaO was the extent of agreement between parenting needs 
reported by parents on the self -appraisal inventory and 
parenting needs of parents as seen by project staff ? 
The purpose of question three is to obtain some estimate of the value 
of a self -report assessment in planning parent training activities. Project 
^ staff will complete the Parenting Self-Appraisal Inventory for each parent. 
The results, of the inventoried completed by staff and parents will be 
compat'ed for each parent. Agreement between the two inventories will be 
scored according to an objective scheme, and parents may be ranked according 
to extent of agreement with project staff. One should note that there 
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is no Implicit assimption that appraisals completed by project staff 
are necessarily valid* The delf-appraisal inventory i^^ in a developmental 
state, and questions two and three primarily yield process evaluation 
information. . ^ 

4. What were the ob-jectives spec ifi ed in MAPs for parents? 
The response to question four will be a simgle tabulation and 
listing of the objectives specified in the WAPs. 

5/ What was the success rate in completing MAPs? 
This, information is of considerable interest since it is relevant 
to the feasibility of the MAP procedure with parents and also gives 
sorte description of the treatment received by parentfs. Analysis will 
consist of determining the number of objectives specified and completed 
for, each parent. Overall descriptive statistics for parents will be 
provided. 

6. What progress was made in meeting identified needs of parents ? 

There will be two basic approache.s used in responding to question 
six. The fitst will be a comparison of , pre-post responses of parents 
on the Parenting Self-Appraisal Inventory. Parents will respond to the 
pretest inventory within a few weeks after their child enters che 
project, and postt;esting will occxir during May, 1977 for those parents 
whose date of pretest was on or before November 15, 1976. The second 
approach will be a comparison of pre-post responsfes of parents to a 
series of short . objective tests built around areas addressed in the 
cooperative plans of parent training activities. These short objective 
tests are termed the Parent Mini Tests and will, be administered by 
project staff on an individual basis duriTig the home visits. The 
testing schedule will be integrated within each parent's training plan. 
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Analysis of the pre'-post Inventory responses will consider 

changes in the reported areas pf strength and weakness* No formal 

.* ■ 

statistical tests will be computed*^ Analysis of the pre-post 
Parent Mini Tests will test the significance of change by means of 
'repeated-measures ANOVA and the F-test, where the dependent variable, 
will be the sunmative composite of all j^ests given througb>ut the 
year. It is assumed that most or all parents will respond to the 

same test items. One should note that parent recepciveness will 

\ 

affect the value of the Parent Mini Tests, and the approach may 
tjoU be feasible with some parents. \ 

Question six does not conjxder the need for a control ccmparisbn > 
as was the case in the children component. The rationale- V' c/aluating 

IT 

the parent component is that one would not expect; "Lmprovement in the 
absence of project intervention. The- basic assumption is that in 
alnost all cases nhere would be little opportunity for parents to 
acquire the knowledges and skills available through Project KIDS* 
7. What was the reaction of parents to the project? 
' Parents will respond anonymously to a short questionnaire (Parent 
Reaction Questionnaire) in order to^obtain a measure of parent reaction 
to the project. Project staff will introduce the questlonnaii'e to 
parents during a home vikit, and parents will be asked to complete 
and mail the questionnaire to the project office after the hone visit 
is completed. The questionnaire will address reaction to both the 
c apprehensive assessment of the child and services being delivered. 
Parents will respond to the questionnaire during the week of February 
1977. The data collection will exclude those parents whose children 
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did not enter the project: before December 1, 1976. Analyses will 
consist of the computation of the range and average for each 
questionnaire Item, as well as the percent of parents giving favorable- 
unfavorable responses* A second data collection may be scheduled for 
June, 1977 If desired. 

8. What was t he extent of aypeement between parental expectations 
of the child and expecta;^:lons of the child as judged realistic 
by professional personnel? 
Parents will respond to a social expectancy scale (Parental 
Expectation Test) In order to obtain a measure of parent expectations 
about the child's eventual Integration Into normal society. Project 
staff will ulsb respond to the scale independently of parents. An 
F-test for correlated groups will be used to compare the overall 
responses of parents to project staff. If appropriate, parents will 
be divided into realistic and unrealistic expectors, and a search 
for concomitant variables such as educational level will be made. 
Data collection will take place during January, 1977. 

. 9 . What -were thfe observed changes in parental needs over a 

' three'»year period? . 
There can be no response to question nine during the 1976-77 
year, but provision for longitudinal study needs' to be made (see 

0 

question ten in childrei. component). 

i 

10. What was the cost of the parent ccmp.^nent? 
The response to this question will use essentiiilly the same 
procedure outlined under question^eleven in the ciiAlldren component. 
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Scaff Pavel Qpcnent Ccmpone nt. 

Project Procedure > The primary thrust of the s^taff davelopnent 
component is the identification of knowledges and skills and the , 
development of suitable staff training sessions to maintain and 
augcaent existing levels of competency among project staff. The 
Staff Development Survey instrument is being developed for use as a 
baseline document with. appropriate training sessions and activities. 
The primary areas of staff development are childhood development, 
assessment of multiply handicapped, educational progransoing, and 
parent involvement. 

Evaluation Questions. 

1. What were the staff competencies identified by project staff? 

2. What was the formal training received by staff in Project KIDS? 

3. What was the observed improvement in staff competencies? 

4. What was the cost of the staff development component? 

Deaign and Analysis. The following details evaluation methodology 
for each question, 

1. What were the staff competencies i dentified by project staff? 
The information source for question one will be the competencies 

and skills which will comprise the Staff Development Survey instrument. 
The infoiaaatiou wi.ll not only be fundamental to Project KIDS but 
should also be useful to similar projects, especially those in large 
urbaii settings. 

2. W hat was the formal training received by staff in Project KIDS? 
Project staff will maintain- a log of all staff development activitie 

and analysis will consist of a descriptive summary of these activities. 
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3. W hat was the observed Improvement In staff competencies? 
The basic design, for responding to question three will be a pre- 
post observation with the Staff Development Survey. Demonstration of 
posttest competencies and skills will likely occur on a testing plan 
individualized for project staff. Appropriate statistical analyses 
will be computed to test the significance of any observed pre-post 
changes (one should note that the use of the pre-post design depends 
upon successful developnent of the survey instirxment). 

An obvious difficulty in responding to question three is that 
the pretest measure of competency needs to be take^ prior to completed 
development of the Staff Development Survey. Existing instrxments 
for surveying staff competencies are currently being reviewed. Once 
a suitable instrument has be^n selected, a J^retest will be sivtn n 
order to collect baseline data as soon as possible. Posttest data 
will also be collected then with the above instrument. 

4. What was the cost of the staff ca velopment component? 
The response to this question will use essentially the same 
procedure as outlined under question eleven in the children componant. 
Demonstration/Dissemination Component 

Project Procedure. One of the key features of the D/D component 
is the development and distribution of information packages on selected 
aspects of the project. The underlying logic of this approach is that 
it gives consumers the option of picking and choosing ideas and proce- 
dures which may be* especially suitable to their unique situations. It 
also has the advantage plutting the entire scope of the project into 
more manageable and understandable units. 
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The D/D^ effort of Project KIDS also strives to communicate 
with important "^ci vie and. professional leaders as ^rell as r.oinciuniCy 
groups and pro^ Visional organizations. However, the project must 
necessarily restrict D/D activlniesU^v^ '^^^ are most compatible 

with project operations and are most lik^iy to be productive in tclrms 
of project D/D. The Project KIDS Advisory Board, which includes 
both parents and professional leaders, lends valuable assistance to 
the D/D effort of the project:. 

Evaluation Questions. 

1* /What were, the D/D activities of the project? 

2. What critical information about the project was sought by 
deci si on-nnaker s ? 

3. What was the cost of the D/D component? 

Design and Analysis ^ The following details the evaluation 
procedures for each evaluation question. 

1. What were the D/D activities of the project? 
P-roj4ct> staff wi.ll maintain records of . all D/D activities. 

scheduled and completed from June 1, 1976 to May 31, 1977. Analysis 
will take the form of a descriptive summary of the D/D activities for 
the specified time period. 

2. What critical information about the project was sought by 
de ci si on^naker s ? 

In order to demonstrate and disseminate In any kind of effective 
fashion, one must know the expectations,;' concerns, and perspectives of 
those people who affect the future of the project. The response to 
the above question will strive to identify the expectations, concerns, 
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and perspectives that will impact the project* Data collection will 
consist of informal observational techniques, and analysis vlll 
consist of a descriptive summary. Data collection will ha periodic, 
and there will be a mid-year and final reporting of information, 
3. What was the cost of the D/D component? \ 

the response to this question will use essentially the same 

/" * • 

procedure as outlined under question eleven in the children ccxuponent.;. 

Instrumentation ' . ^ ^ 

The following provider a brief description of each of the measure- 
ment instruments specified in the evaluation. - 
Bayley Scales of Infant Development ; 

The Bayley is one of the best known instruments for measuring 
developmental status in the first two and one-half yefars of life. 
The instrument yields a Mental Development Index, a Psychomotor ^ . 
Development Index,, and a Behavior Record. The standardization sample 
included 1,26'2 children and was representative of the United States 
population (1960 census) in* terms of urban-rural residence, white- 
nonwhite race, occupation and education of the head of the ^housei^old, 
and geographic region (of course, the standardization sample included 
only normal children). Split-half reliability; coefficients re^iarted 
for the mental scale range from .81 to .93, and coefficients for the 
motor scale range from -.68 to .92. 
McCarthy Scales of Children's Abilities ; 

The McCarthy Scales were designed to assess strengths and weaknesses 
in important abilities of children aged 2h to Qh years (McCarthy, 1972). 
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The total test includes 18 subtests whi^gh combine to form six scales: s 
verbal, perceptual •performance, quantitative, general cognitive, 
•memory, and motor* The standardization ss^aple Included 103.2 normal 

"* ■ ^ ■• - 

children and was representative of the United States population in 
terms of sex, geographic region, race, .and father's occupation. 

Only the perceptual-performance scale and the motor scale will be 
^ used in the evaluation of the project* It is thought that these two 
scales are moat pertinent to project goals* - The reporteV^eiiability 
of the perceptual -performance scale ranges from 175 to .90, and the - 
reliability of the motor scale ranges from .60 to •84. It may be 
necessary to modify the administration selected test items to 
suit special needs of children in Project KIDS. If so, the testing 
procedure will be standardized for the pre<*post measurement, and caution 
will be' exercised when referencing test norm^. 
KIDS Inventory of Development Scale t 

The KIDS Inventory is in a developmental status and is tmdergoing 
further refinement and mpdif ication. The basic format of the Inventory 
is a checklist of developmental behaviors sequenced according to 
chrotiological age 0 to 72 months in four areas of development: gross 
motor, fine motor, language/ cognitive, and self-help* Scoring of 
the inventory* is pass-fail for each behavior tested. 
KIDS Parenting Self-Appraisal Inventory (PSAIr) ; * 

. The PSAI is in a dex^elopmantal status* The besic structure of 
the instrument is a checklist of a sample of basic knowledges pertinent 
to the parenting of developmentally delayed children. The sampled 
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areas (and items within areas) represent priorities established by 
project staff and include such areas as mental deficiency, sequential 
growtH and development, public school special education, nutrition, 
and child management. ♦ Parents respond to vthe PSAI by checking those 
knowledges in which they feel strong or weak. 
KIDS Parent Mi ni Tests (PMT) : . v 

The PMT will be deisigned 'throughout the year as parents address 
sub-areas within the KIDS Parenting Self -Appraisal Inventory. Each 
PMT will consist of up to ten true-false items designed to measure 
knowledge acquisition as programmed (either directly or indirectly) 
in the parent Mini Actl^/lty Plans. Most PMTs will be given orally 
by project "staff in the home. The total PMT will be a composite of 
all PMTs ada^inis tared during the year, and reliability ahd^ validity ^ 
data on the total PMT will be forthcoming. 
KIDS Parent Reaction Questionnaire (PRQ) : 

The PRQ will contiain up to' 20 Likert-type items which will solicit 
parent opinion and reaction about selected aspects of the project, such 
as the assessment procedure^ charting the MAPs, project staff, and so 
forth. Parents will respond to the PRQ anonymously. 
KIDS Pare ntal Expectations Test (PET) ; 

The PET will be designed to meagyafe^p^rent expectations abcut 
their child's integration into normal society. The basic structure 
of the test will be a set of normal behaviors performed by. adults and 
school children. Parents will respond to the PET, by indicating the 
extent to which they think their child would be able to perform the 
normal behavior. The test may be administered either orally or in 
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written form, and most will be given by project staff during a home 
visit. Reliability and validity data will be forthcomingo 
KtPS Staff Development Survey (SDS) :? 

The SDS atteccpta to survey the range of knowledges and skills 
needed ^for the successfxil performance of project staff in a large 

urbajtf*^ setting. The basic structure of the instrument will be a 

f " . V • 

sejtf-report checklist format, where Survey items are scored pass-fail. 

ThjB ins truant is still in a developmental state and is undergoing 

refinement and modification. The SDS may be self-administered or 

group t administered as desired, 

\ • 

Data Collection Schedule 
- The evaluation of Project KIDS entails a consi<jlerable amount 
of data collection, some of which is on going, such as objective 
mastery in the MAPi5,uC0St record keeping, and record of demonstration/ 
dissemination activities. However, a number of formal measurement 
processes must occur at specified times during the* 1976-77 year. 
Table 4 lists these instriiments and specified dates for data collection. 

Table 4 

Schedule for Administration of Evaluation Instruments 
^ 



Instrument Date.(s) 



Bayley Scales, of Infant Development 12-month pre-post interval per 

\ child 
McCarthy Scales^f Children's Abilities 

KIDS Inventory of Development 

KIDS Parenting Self rAppraisal Inventory Up through Nov. 15, 1976/May, 1977 

KIDS Parent Mini Tests Before and after each MAP (parent) 

KIDS Parent Reaction Questionnaire Feb. 28-March 4, 1976/June, 1977 

(Optional) 

KIDS Parental Expectations Test January, 1977 

KIDS Staff Development Survey As developed/ June, 1977 
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' Reporting Schedule 
There will be two major fomatl reporting dates for the evaluation 
of Project KIDS. The first will be January 14, 1977, and the second 
will be June 30, 1977. Of course, informal reporting will be on a 
continuing basis, but the scheduling of a mid«year formal interim 
report (January 14, 1977) provides a kind of milestone for as-sessing 
project implementation and for considering operational modifications. 
Table 5 lists evaluation questions (in abbreviated form) wl.thin each 
projeCit component and cites formal reporting dates for each question. 



Table 5 

Formal Reporting Dates for 
Evaluation Questions 




r.- 

Question Interim Report 

(January 14, 1977) 


Final 
(Jvme 


Report 
30, 1977) 


ComponenC: Children 






1. 


number of children served X 


X 




2. 


instructional history X 


X 




3. 


demographic characteristics 


X 




4; 


areas of delay in the MAFs 


X 




5. 


instructional objectives X 


X 




6 . 


extent of instruction X 


X 




.7- 


observed improvement in areas of delay 


X 




8. 


time constraints to master objectives X 


X 




9. 


improvetaent due to project intervention 







10« three -year instructional history 

ll. cost of children caiponent ♦ X 
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Table 5 (cont'd) 

Formal Reporting Dates for 
Evaluation Questions 



Question Interim ' Report Final Report 

- ^ ■ (January 14, 1977) • (juna 30', 1977) 



Component : Parent 

lm^''%€iiogvaph±a characteristics - ' X 

2. needs reported by parents X 

3. agreement between parents and staff 

on reported needs X X 

4. objectives in' parent s'.llAPs X X 

5. success rate in completing MAPs X X 

6. progress in parent needs > X 

7. parent reaction to project X X 

8. agreement between parents and staff 

on child .expectations X 

•9. three-year changes in parent needs 

10. cost of parent component ^ . X 

Component:' Staff Development 

- 1. identified ccmpe^ncies * X X 

2. training received by staff X 

3. .Improvement in competencies X 

4. cost of staff component • . X 
Comp on ent z Demons tr a t i on/ Diss emi na t i on 

1. D/D activities / ^ ^ 

2. Critical information X X 

3. Cpst of D/D component X 
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